Spatial-temporal video quality metric based on an estimation of QoE

International audienceIn this work a new Reduced Reference (RR) Video Quality Metric (VQM) is proposed. The method takes advantage of the Human Visual System (HVS) sensitivity to sharp changes in the video. In the first step, the matching regions are calculated using a temporal approach. The quality of these regions are then evaluated using a spatial approach in the last step the quality of the video is calculated based on the parameters gathered in the spatial and temporal domain. An important improvement lies in taking into account the Quality of Experience (QoE) represented as the motion activity density of the reference video. Due to the spatial-temporal approach taken, the metric is named STAQ (Spatial-Temporal Assessment of Quality). The results show a great improvement in the case of H.264 and MPEG-2 compressed and IP distorted videos even when compared to state of the art Full Reference (FR) metrics

Towards a perceptual metric for video quality assessment

In the last couple of years a huge amount of work has been shifted from works on Image Quality Assessment to Video Quality Assessment. Although some metrics have started to take the temporal aspects of videos into account but still most metrics are focusing on the spatial distortions in videos or in other words applying image quality metrics on individual frames. Also till now most metrics are focusing on the Quality of Service (QOS) rather than the Quality of Experience (QOE). With respect to the mentioned factors we believe that there is a need for a new metric which has a Spatial-Temporal approach and takes QOE into account and so the proposed metric is based on these two main approaches. Because of the spatial-temporal approach we had the metric was named as STAQ (Spatial-Temporal Assessment of Quality). Our proposed method is based on the fact that the Human Visual System (HVS) is sensitive to sharp changes in videos. Keeping this in mind we could reach the conclusion that there will be matching regions in consecutive frames. We took advantage of this point and found these regions and used a Full Reference Image Quality Metric to evaluate the quality of these frames. We also used five different Motion Activity Density groups to evaluate the amount of motion in the video. Our final score was later pooled based on five different pooling functions each representing one of the motion activity groups. In other words we used QOE or information from subjective evaluation for playing a controlling factor role in our method. When the proposed reduced reference metric is compared to ten different state of the art full reference metrics the results show a great improvement in the case of H.264 compressed videos compared to other state of the art metrics. We also reached good results in the case of MPEG-2 compressed videos and videos affected by IP distortion. With respect to the results achieved we could claim that the metric introduced is among the best metrics so far and has especially made a huge progress in the case of H.264 compressed videos

Deep Learning in Image Quality Assessment: Past, Present, and What Lies Ahead

Quality assessment of images plays an important role in different applications in image processing and computer vision. While subjective quality assessment of images is the most accurate approach due to issues objective quality metrics have been the go to approach. Until recently most such metrics have taken advantage of different handcrafted features. Similar (but with a slower speed) to other applications in image processing and computer vision, different machine learning techniques, more specifically Convolutional Neural Networks (CNNs) have been introduced in different tasks related to image quality assessment. In this short paper which is a supplement to a focal talk given with the same title at the London Imaging Meeting (LIM) 2021 we aim to provide a short timeline on how CNNs have been used in the field of image quality assessment so far, how the field could take advantage of CNNs to evaluate the image quality, and what we expect will happen in the near future

From regular text to artistic writing and artworks: Fourier statistics of images with low and high aesthetic appeal

The spatial characteristics of letters and their influence on readability and letter identification have been intensely studied during the last decades. There have been few studies, however, on statistical image properties that reflect more global aspects of text, for example properties that may relate to its aesthetic appeal. It has been shown that natural scenes and a large variety of visual artworks possess a scale-invariant Fourier power spectrum that falls off linearly with increasing frequency in log-log plots. We asked whether images of text share this property. As expected, the Fourier spectrum of images of regular typed or handwritten text is highly anisotropic, i.e. the spectral image properties in vertical, horizontal and oblique orientations differ. Moreover, the spatial frequency spectra of text images are not scale invariant in any direction. The decline is shallower in the low-frequency part of the spectrum for text than for aesthetic artworks, whereas, in the high-frequency part, it is steeper. These results indicate that, in general, images of regular text contain less global structure (low spatial frequencies) relative to fine detail (high spatial frequencies) than images of aesthetics artworks. Moreover, we studied images of text with artistic claim (ornate print and calligraphy) and ornamental art. For some measures, these images assume average values intermediate between regular text and aesthetic artworks. Finally, to answer the question of whether the statistical properties measured by us are universal amongst humans or are subject to intercultural differences, we compared images from three different cultural backgrounds (Western, East Asian and Arabic). Results for different categories (regular text, aesthetic writing, ornamental art and fine art) were similar across cultures

Quality is in the Salient Region of the Image

The ultimate goal in any proposed Image Quality Metrics (IQMs) is to accurately predict the subjective quality scores given by observers. In the case of most IQMs the quality score is calculated by pooling the quality scores from what is referred to as a quality map of an image. While different pooling methods have been proposed, most such approaches use various types of a weighting average over the quality map to calculate the image quality score. One such approach is to use saliency maps as a weighting factor in our pooling process. Such an approach will result in giving a higher weight to the salient regions of the image. In this work we study if we can evaluate the quality of an image by only calculating the quality of the most salient region in the image. Such an approach could possibly reduce the computational time and power needed for image quality assessment. Results show that in most cases, depending on the saliency calculation method used, we can improve the accuracy of IQMs by simply calculating the quality of a region in the image which covers as low as 20% of the salient energy

Reviving Traditional Image Quality Metrics Using CNNs

Objective Image Quality Metrics (IQMs) are introduced with the goal of modeling the perceptual quality scores given by observers to an image. In this study we use a pre-trained Convolutional Neural Network (CNN) model to extract feature maps at different convolutional layers of the test and reference image. We then compare the feature maps using traditional IQMs such as: SSIM, MSE, and PSNR. Experimental results on four benchmark datasets show that our proposed approach can increase the accuracy of these IQMs by an average of 23%. Compared to I I other state-of-the-art IQMs, our proposed approach can either outperform or perform as good as the mentioned I I metrics. We can show that by linking traditional IQMs and pre-trained CNN models we are able to evaluate image quality with a high accuracy

Image Quality Assessment by Comparing CNN Features between Images

Finding an objective image quality metric that matches the subjective quality has always been a challenging task. We propose a new full reference image quality metric based on features extracted from Convolutional Neural Networks (CNNs). Using a pre-trained AlexNet model, we extract feature maps of the test and reference images at multiple layers, and compare their feature similarity at each layer. Such similarity scores are then pooled across layers to obtain an overall quality value. Experimental results on four state-of-the-art databases show that our metric is either on par or outperforms 10 other state-of-the-art metrics, demonstrating that CNN features at multiple levels are superior to handcrafted features used in most image quality metrics in capturing aspects that matter for discriminative perception

Preventing Over-Enhancement Using Modified ICSO Algorithm

This paper proposes an Image Contrast Enhancement (ICE) method based on using an Improved Chicken Swarm Optimization (ICSO) algorithm to enhance images while at the same time preventing over-enhancement. In the optimization process, a new practical objective function is employed to reach three main goals, preserving the main details, generating an image with a uniform histogram, and reducing the spikes in the modified histogram. In the proposed approach, the RGB color channels are optimized individually. The performance of the proposed method is suitable for enhancing the contrast of low- and high-contrast images. A subjective experiment is designed to visually evaluate and compare the results with other ICE methods. The simulation results on the CSIQ, TID2013, and SEID datasets show that the proposed method outperforms numerous traditional and state-of-the-art ICE techniques both subjectively and objectively. The most important advantage of the newly proposed technique is that there is an agreement among observers on when over-enhancement occurs regardless of whether the Initial processed image was of low or high contrast

How Good is Too Good? A Subjective Study on Over Enhancement of Images

For a long time different studies have focused on introducing new image enhancement techniques. While these techniques show a good performance and are able to increase the quality of images, little attention has been paid to how and when overenhancement occurs in the image. This could possibly be linked to the fact that current image quality metrics are not able to accurately evaluate the quality of enhanced images. In this study we introduce the Subjective Enhanced Image Dataset (SEID) in which 15 observers are asked to enhance the quality of 30 reference images which are shown to them once at a low and another time at a high contrast. Observers were instructed to enhance the quality of the images to the point that any more enhancement will result in a drop in the image quality. Results show that there is an agreement between observers on when over-enhancement occurs and this point is closely similar no matter if the high contrast or the low contrast image is enhanced